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Abstract 

Given a matrix order n, a restart parameter m (m < n), a decreasing positive sequence /(0) > 
/(l) > . . . > f(q) > 0, where q < n/m, it is shown that there exits an n-bj-n matrix A and a vector 
ro with ||ro|| = /(0) such that ||rfc|| = f(k), k = 1, . . . ,q, where rt is the residual at cycle k of restarted 
■ GMRES with restart parameter m applied to the linear system Ax — b, with initial residual ro = b — Axo. 

Moreover, the matrix A can be chosen to have any desired eigenvalues. We can also construct arbitrary 
cases of stagnation; namely, when /(0) > /(l) > . . . > f(i) = f(i + 1) > for any i < q. The restart 
' parameter can be fixed or variable. 

^ ■ 1 Introduction 

We consider the generalized minimal residual method (GMRES) [15j for solution of a nonsingular non- 
Hcrmitian systems of linear equations 

Ax = b, AeC nxn , beC n . (1) 

C*~) \ For a few class of matrices, some convergence estimates are available for restarted GMRES and full 

GMRES. For example for real positive definite matrices (that is, for matrices A for which H = (A + A H )/2 
is symmetric positive definite, or, equivalently, for matrices A for which x H Ax > for any nonzero x € R"), 
the Elman's bound [H [3 |TTJ US] can be stated as follows 

O; ||r fc || 2 < (l-p) fe ||r || 2 where < p = (A mm (H)/|| A||) 2 < 1. 

ON . 

The latter guarantees linear convergence of GMRES(m) for any value of in > 1 for a positive definite matrix. 
Improvements and generalizations of this bound can be found in [3l I16[ I21j . 

For normal matrices the convergence of both full and restarted GMRES is well studied. In particular, the 
convergence of full GMRES for normal matrices is known to be linear and there exist convergence estimates 
governed solely by the spectrum of A (TTJ [TS] . The convergence of restarted GMRES for normal matrices, 
however, is sublinear [2][19]. The current paper is concerned with the general case. 

For the general case, the following theorem proves that we can not prove convergence results based on 
the spectrum of the coefficient matrix alone. 

Theorem 1 (Grccnbaum, Ptak, and Strakos, 1996, |12j ) Given a nonincreasing positive sequence /(0) > 
/(I) > ' ' ' ^ f( n ~ 1) > 0; there exists an n-by-n matrix A and a vector r with \\r \\ = /(0) such that 
f{k) = \\ru\\, k = 1, . ..,n— 1, where r fc is the residual at step k of the GMRES algorithm applied to the 
linear system Ax = b, with initial residual r = b — Ax . Moreover, the matrix A can be chosen to have any 
desired eigenvalues. 

This result states that, in general, eigenvalues alone do not determine the convergence of full GMRES. 
Assuming that the coefficient matrix A is diagonalizable, some characterizations of the convergence of full 
GMRES rely on the condition number of the eigenbasis [TH] . Other characterizations of the convergence of 
full GMRES rely on pseudospectra [14]. More commonly, the field of values is used [3l l6l 171 [TT1 H5l [TBI [21] . 
A discussion on how descriptive some of these bounds are is given by Embree [8] . 
The main result of this paper is given in the abstract. We will repeat it here 
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Theorem 2 Given a matrix order n, a restart parameter m (m < n), a decreasing positive sequence /(0) > 
f(l) > . . . > f(q) > 0, where q < n/m, there exits an n-by-n matrix A and a vector Tq with \\ro\\ = /(0) 
such that \\rk\\ = f(k), k = 1, . . . ,q, where r^ is the residual at cycle k of restarted GMRES with restart 
parameter m applied to the linear system Ax = b, with initial residual vq = b — Axq ■ Moreover, the matrix 
A can be chosen to have any desired eigenvalues. 

Section [5] contains a proof of Theorem [51 Theorem [2] is to restarted GMRES what Theorem [1] is to full 
GMRES. The proof we provide is constructive and directly inspired by the article of Greenbaum, Ptak, and 
Strakos [12]. Although Greenbaum, Ptak, and Strakos laid the path, there arc several specific difficulties 
ahead in the case of restarted GMRES. 

Full GMRES has a nonincreasing convergence (for any i > 0, f(i) > f(i + 1)) and it computes the 
exact solution in at most n steps (f(n) = 0). It is remarkable that Greenbaum, Ptak, and Strakos are 
able to characterize any admissible convergence for GMRES. (See assumptions on / in Thcorcm[T]) At the 
same time we would like to note that the cycle-convergence of restarted GMRES can have two admissible 
scenarios: either for any i, f(i) > f{i + 1), in other words, the cycle-convergence is decreasing; or there exits 
s such that f(i) > f(i + 1) for any i < s, and then for any i > s, f{i) = /(s), in other words, if restarted 
GMRES stagnates at cycle s + 1, it stagnates forever. Theorem [5] considers the first case (decreasing 
cycle-convergence). In Section [3J we consider the second case (stagnation). Therefore with Theorem [2] and 
Section [31 we prove that any admissible cycle-convergence curve is possible for the q first cycles of restarted 
GMRES. 

As mentioned above, the maximum number of iterations of full GMRES is at most n, and the method 
delivers the exact solution in a finite number of steps. Restarted GMRES, however, may never provide the 
exact solution. It will (hopefully) decrease the residual norm at each cycle, that is, provide a more and more 
accurate approximation to the exact solution. With n 2 parameters in A and n parameters in b we are not 
able to control the convergence for an infinite amount of cycles. For this reason, it is natural to consider only 
the first q < n/m initial GMRES(m) cycles. Actually, we provide the same level of control as Greenbaum, 
Ptak, and Strakos: n iterations (or q cycles with q < n/m) and n eigenvalues. 

In Section [U we generalize the result given by Theorem [2] and Section [3] for the case of variable restart 
parameters. The sequence of restart parameters m^ needs to be known a priori. We show that GMRES(?7ifc) 
can produce any admissible cycle-convergence curve at the q initial cycles, regardless of the spectrum of the 
coefficient matrix, where q is such that 2<=i TO fc < n - We note that our construction can be a reasonable tool 
for generating examples/counter-examples for different strategies for varying the restart parameter, e.g. [2]. 

The cycle-convergence of restarted GMRES for normal matrices is sublinear [21 [19]. However, for general 
matrices, through Theorem^ one can expect any convergence curve. In particular, it is possible to construct 
matrices for which the convergence of GMRES (m) is fast (e.g. superlincar). This relates to the observations 
of Zhong and Morgan |20j who report superlincar cycle-convergence for their particular cases of nonnormal 
matrices, as well as to [19] . where it is shown that the cycle-convergence can become superlinear as the 
coefficient matrix departs from normality. 

In a pedagogical paper, Embree [5] presents a 3-by-3 linear system of equations and attempts to solve it 
with GMRES(l) and GMRES(2). While GMRES(l) converges to the exact solution in 3 cycles, GMRES (2) 
(almost) stagnates. Our main result, basically, reaffirms this intuition in the sense that the increase in the 
restart parameter (and thus, in the computational complexity at each cycle) does not necessarily imply a 
faster convergence. 

In order to improve the convergence of restarted GMRES, several techniques [U HI El UHl H3] have been 
proposed which consist of augmenting (or enriching) the Krylov space with eigenvectors or, alternatively, 
deflating some of the eigenvalues from the spectrum of the original matrix A. The eigenvalues targetted 
are the ones the closest from zero. These techniques have proved effective and the convergence of restarted 
GMRES is, in practice, greatly improved. Theorem [2] states that, in the general case, eigenvalues alone do 
not determine the convergence of restarted GMRES, therefore it is hard to provide a theorical justification 
for the choice of removing the eigenvalues the closest from zero. A beginning of theoretical understanding 
has been provided by Zitko [2"2"] , 

We have generated two Matlab functions that correspond to Theorem [T] and Theorem [2] Given a matrix 
size, a restart parameter, a convergence curve and a spectrum, we construct the appropriate matrix and 
right-hand side. See: http://www-matli.cudenver.edu/~eugenev/edf.software/caiycurve/. 



2 



The main message that we would like our readers to retain from this paper is that in the context of 
GMRES(m), for a certain number of initial cycles, any convergence curve is possible independently of the 
spectrum of the coefficient matrix. This means that eigenvalues alone do not determine the convergence of 
restarted GMRES. 

2 Constructive proof of Theorem [2] 

Let n be a matrix order and m a restart parameter (m < n), A = {Ai, A2, . . . X n } C C \ {0} be a set of n 
nonzero complex numbers, and {f(k)}1 =0 be a decreasing sequence of positive real numbers, q < n/m. 

In this section we construct a matrix A G c nxrl anc i an initial residual vector ro = b — Axq G C" such 
that GMRES (m) applied to the system (TTJ) with the initial approximate solution xq, produces a sequence 
{ x k\\ = \ of approximate solutions with corresponding residual vectors {rk}1 =0 having the prescribed norms: 
= /(&)• Moreover the spectrum of A is A. 

2.1 Outline of the proof 

The general approach described in this paper is similar to the approach of Grecnbaum, Ptak, and Strakos [12] : 
we fix an initial residual vector, construct an appropriate basis of C" and use this basis to define a linear 
operator A. This operator is represented by the matrix A in the canonical basis. It has the prescribed 
spectrum and provides the desired cycle-convergence at the first q cycles of GMRES (to). However, the 
presence of restarts somewhat complicates the construction: the choice of the basis vectors, as well as 
the structure of the resulting operator A, becomes less transparent. Below we describe our three-step 
construction. 

At the first step we construct q sets of vectors Wm = { w [ k \ ■ ■ ■ > Wm }, k = 1, . . . , q, each set wj? is the 

orthonormal basis of the Krylov residual subspace AJC m (A,rk-i) generated at the fc-th GMRES(to) cycle 
such that 

span wj fc) = AJCj (A, r fc _i) . ./ 1 m. (2) 

(With this definition, Wm' is defined up to multiplication by a complex number of unit modulus.) 

The orthonormal basis Wm' needs to be chosen in order to generate residual vectors r^ with the prescribed 
norms f(k) at the end of each cycle subject to the additional requirement that the set of mq + l(< n) vectors 

"C r (!) (!) ( 2 ) (2) (9) (?) 1 lo\ 

is linearly independent. 

Once we have the set S, we will complete it to have a basis for C™. When the number of vectors in S is 
less than n, a basis S of C™ is obtained by completion of S with a set iS of n — mq — 1 vectors, i.e. S = {S, S}. 
This will provide a representation of C n as the direct sum 

C" = span S = spanjrcW^!} © • • ■ © span{r 9 _i, W^J © span{r g ,<S}. (4) 

The latter translates in terms of Krylov subspaces into 

C" = span S = K m (A, r ) © • • • © JC m (A, r 9 _i) © span{r g , <S}. 

At the second step of our construction, we define a linear operator A : C" — ► C" with spectrum A 
which generates the Krylov residual subspaces in Eq. ([2]) at each GMRES(to) cycle, by its action on the 
basis vectors 5, such that the desired matrix A is the operator ^4's representation in the canonical basis. 
The third step accomplishes the construction by a similarity transformation. 

The two following subsections are concerned with the question if ((2])-([3])-([4|) and the definition of the 
operator A with the prescribed spectrum is actually possible. 
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2.2 Step 1: Construction of a sequence of Krylov subspaces which provide the 
prescribed cycle— convergence 

At the fc-th GMRES(m) cycle, the residual vector r k satisfies the following minimality condition: 

IK || = mm JK_i-k||. (5) 

We assume that each set W^' is an orthonormal basis of a corresponding Krylov residual subspace 
AJC m (A, rk-i), therefore the condition (JSJ) implies 

m 

rfc = rfc-i - y^K-i, wf^wf 1 ^, k = l,...,q. (6) 
i=i 

At this stage, in order to simplify the forthcoming justification of the linear independence of the set S, 
we impose a stricter requirement on the residual change inside the cycle. We will require that the residual 
vector rfc_i remains constant during the first m — 1 inner steps of GMRES and is reduced only at the last, 
m-th, step. Thus, the equality in ([6]) can be written as 

r k =r k _ 1 -(r k _ 1 ,w$)w<$, k = l,...,q. (7) 

(fe) 

j 

(r fc _i,wj fc) } = 0, j = l,...,m-l, k = l,...,q. (8) 
From Eq. ([7]), using the fact that r k J- Wm and the Pythagorean theorem, we obtain 

\(r^ u w^)\ = ^\\r k ^-\\r k \\^ k = l,...,q. 

We rewrite the expression above in terms of cosines of angles tp k = Z(r k ^i, Wm^) by prescribing the expected 
values f(k) for the norms of the residuals. We get 



This implies that the vectors Wj , j = 1, . . . , m — 1, are orthogonal to the residual vector r k _\, i.e 



cosVjfe = #jns e(0il)i *=i «. o) 

- 1) 

This latter equation means that, if we are given r k -i, one way to ensure the desired cycle-convergence 
at cycle k of GMRES(m) is to choose the unit vectors such that ((7])-((9|) holds. 

In the following lemma, we show constructively that the described approach (JTJ)— ([9]) leads to an appro- 
priate set S. 

Lemma 1 Given an initial vector ro, ||ro|| = f(0), there exist vectors r k , \\r k \\ = f(k) and orthonormal sets 
Wm^ such that Eq. (0), and hold, and the set S is linearly independent, k = 1, . . . , q < n/m. 

Proof. The proof is by induction. 

Let k = 1. Given the initial vector ro, ||ro|| = /(0), we pick W^ij — {u>i , . . . , w^Li} an orthonormal 
set in rjj- in order to satisfy Eq. ([5]). The set {ro, VV^^} is linearly independent. 

In order to choose the unit vector Wm orthogonal to the previously constructed vectors W < ^ > _ 1 and 
satisfying Eq. we introduce a unit vector y^> £ { r 0iWmli} 1 > so that 

w m = yr^y 008 ^ 1 + 2/ (1)si W , i- 

We find the vector n by satisfying Eq. jTJ). Eq. J9]) guarantees that ||ri|| = /(l), as desired. Finally we 
append the constructed vector r\ to {r , W^_ x } and get the set {r , W^i_ 1; r{\, which is linearly independent, 
since, by construction, r\ is not in span {ro, W^ii}. 
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The induction assumption is that we have constructed k — 1 vectors r±, . . . ,rk-i with the prescribed 
norms /(l), . . . , f(k — 1) and orthonormal sets Wm^ ,.. ., Wm , such that the equalities ([7]), © and © 
hold, and the set 

{r , Wi^ l7 . . . , r k - 2 , W£l?,r k -i} (10) 
is linearly independent. We want to show that we can construct the next vector r k , \\r k \\ = f{k), and the 
orthonormal set Wm' , satisfying Eq. ([7]) , ||SJ) and © , such that 

{r , W^ 1; . . . , r k - 2 , W£zt\ rjb-i, w£l 1} r fc } (11) 

is linearly independent, k < q. 

We start by constructing orthonormal vectors W„lj = , ■ • • j^m-i}' satisfying Eq. §8§, with the 
additional requirement that the set W^-i ^ s n0 ^ m ^ ne span of the previously constructed vectors given in the 
set ((TO)) . From these considerations we choose W^-i as an orthonormal set in the orthogonal complement 
of *m i.e. 



w 



{k) G {r , , . . . , r fe _ 2 , W^Zi ,r k -i} x , j = 1, . . . , m - 1. 



Appending W^l x to the set (JTUJ) will give a linearly independent set. 

■ 



To finish the proof, we need to construct the vector w^} , satisfying Eq. © and orthogonal to W„ 



For this reason we introduce a unit vector y^ k \ 

e {r ,wl 1 i ll ... 1 r M! )vt" 1 Vn,wi ft l 1 } i 1 

so that Wm 

w ™ } = f[k~- \) C0S ^ k + ^ (fc)sin ^ fc ' 
We define the vector r k with Eq. ([Jj. Eq. ([9]) guarantees = /(/c). The set (fTT|) is linearly independent, 
since, by construction, the vector is not in span {r , W„l l5 . . . , rfe_2, WjjJiTi , Ffc-i, W^ljJ. 

□ 

2.3 Step 2: Definition of a linear operator with any prescribed spectrum 

So far we have shown that, given an initial residual vector ro, ||ro|| = /(0), it is possible to construct vectors 

r k, \\ r k\\ = f(k), and orthonormal vectors wi , k = 1, . . . ,q, satisfying Eq. (UJ, ([5]) and ©, such that the 
set iS of mq + 1 vectors is linearly independent. 

In order to define a unique linear operator, we need to have a valid basis of C" on hand. Thus, we expand 
the set S by linearly independent vectors S = {si , . . . , si } , t = n — mq — 1 : 

S = {r , W^, . . . , r 9 -i, W^.r,, «i, ...,**}, (12) 

so that 5 is a basis of C" . 

Before we define a linear operator A, let us consider the set A = {Ai, A2, . . . , A„} of nonzero numbers in 

the complex plane that will define A's spectrum. Wc split A into q + 1 disjoint subsets 

A = {Ai,A2,...,A g ,A 9+ i}, 

such that each A k , k = I, . . . ,q, contains m elements of A, and the remaining n — mq elements are included 
into Ag_|_i. 

For each set A^ we define a monic polynomial p k (x), such that the roots of this polynomial are exactly 
the elements of the corresponding A^: 

m— 1 



p k (x) = x m -J2 a ? )xj , k = l,...,q; (13) 
t 

p q +i(x) = x t+1 -J2 a j 9+1)xj > t = n - mq - 1; (14) 



3=0 
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(15) 



with otj's being the coefficients of the respective polynomials, ^ 0, k = 1, . . . q + 1. Pk(x) can 

be considered as the characteristic polynomial of an m-by-m matrix with spectrum A&. p q +i(x) can be 
considered as the characteristic polynomial of a (t + l)-by-(t + 1) matrix with spectrum A 9+ i. 
We define the operator A : C" — ► C" as follow: 

Ar k -i = w[ k \ 

Aw™ = w (k \ 

A W ( k ) i ( fc ) , ( k ) ( fc ) , , ( fc ) ( fc ) ; i 

-4™m-i = -"o r k + a a 'r k -i + oc\ w\ H h oW-i^li, & = 1, • • •<?; 

„4r g = sx: 

-4Si = S2, 

-4S"t-i = St j 

^ = a ( q+1) r q + a<f +1) s 1 + --- + al q+1) st, 

where ct!y 's are the coefhcients of polynomials (|T3)) and (fT4]) . 

The following lemma shows that, given vectors r& and orthonormal sets Wm constructed according to 
Lemma [TJ the linear operator A, defined by (|15|) and represented by a matrix A in the canonical basis, 
generates the desired Krylov residual subspaccs given in Eq. and the spectrum of A can be arbitrarily 
chosen. 

Lemma 2 Let the initial residual vector rg, ||ro|| = /(0), as well as the residual vectors r k and orthonormal 
sets Wm^ be constructed according to Lemma\^ Let S be the basis o/C" as defined by Eq. 112^1. We assume 
a matrix A to be the representation in the canonical basis of the linear operator A defined by Eq. t!5\) . 
Then the linear operator A generates the Krylov residual subspaces given in Eq. Moreover, A has the 
prescribed spectrum A. 

Proof. Directly from the definition (fT5|) of the linear operator .4., for k = 1, . . . , q, we have: 
span{^r fc _i, . . . , A 3 r k ^i) = span wf\ j = 1, . . . , m - 1. 

To see that, for each k, 

span{.4r fe _i, . . .,A m r k -i} = span W%\ 

notice that, by Eq. l[7|). (— q;q r k +a^r k ~i) £ span{u)m }• Thus, given the representation A of the operator 
A in the canonical basis, Eq. ([2]) holds for each k, k = 1, . . . , q. 

To prove that the arbitrarily chosen set A is the spectrum of A, let us consider the matrix [A] s of the 



ti 



operator A in the basis S: 






■• 


• a (1 

"0 


1 


•• 


(i 





1 •• 


<*2 







m—1 











(2) 

o 

(2) 
1 

(2) 
2 



(2) 



,(2) 



>0 






• 


• a iq+1 


1 


■ 







1 • 


• a^ +1 







.,(9+1) 



The matrix [A] s has a block lower triangular structure, hence [-A] 5 's spectrum is the union of the 
eigenvalues of all diagonal blocks, which are the companion matrices corresponding to the sets with 
characteristic polynomials defined in (fl3|) and (fl4)) . Thus, the spectrum of A is A. 

□ 



2.4 Step 3: Conclusion of the proof of Theorem [2] 

Finally, wc define A as the representation of the operator A in the canonical basis: {ei, C2, ■ ■ ■ , e„}, 



A = S[A] s S-\ (17) 

where the square matrix S is formed by the vectors given in Eq. (|12[) written as columns and [A] s is defined 
by Eq. (|16[) . The constructed matrix A provides the prescribed norms of residual vectors at the first q 
GMRES(m) cycles when starting with rg and its spectrum is A. 



2.5 Difference with the work of Greenbaum, Ptak, and Strakos |12j 

For the reader familiar with the work of Greenbaum, Ptak, and Strakos [12], it might be tempting to 
obtain the present result by pursuing the following scheme: fix ro and then consider the first restarted 
GMRES cycle as the initial part of a full GMRES run where the convergence is prescribed for the first 
m iterations (and arbitrarily set for the remaining n — m iterations). Then, similarly, given the starting 
residual vector r\ provided by this first cycle, construct the next Krylov residual subspace which provides 
the desired convergence following the scheme of Greenbaum, Ptak, and Strakos [12]. Proceed identically for 
the remaining cycles. This approach, however, does not guarantee the linear independence of the set S and, 
hence, one meets the problem of defining the linear operator A. These considerations were the reason for 
the assumption j7|) on the residual reduction inside a cycle, which allowed to quite easily justify the linear 
independence of the set S and, as well, to control the spectrum. 
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3 Generating stagnating example of restarted GMRES 



Theorem [2] handles the case for the decreasing positive sequence {f(k)}1 =0 . In this section, we are concerned 
with the stagnation case: when /(0) > /(l) > • • • > f(s) > and f(s) = f(s + 1) = . . . = /(<?). 

Theorem 3 Given a matrix order n, a restart parameter m (m < n), a positive sequence {f(k)}1 =0 , which 
is either decreasing, or such that f(0) > /(l) > • • ■ > f(s) > and f(s) = f(s + 1) = . . . = f(q), where 
q < n/m, s < q. There exits an n-by-n matrix A and a vector ro with ||ro|| = /(0) such that \\rk\\ = f{k), 
k = 1, . . . , q, where r k is the residual at cycle k of restarted GMRES with restart parameter m applied to the 
linear system Ax = b, with initial residual ro = b — Axq. Moreover, the matrix A can be chosen to have any 
desired eigenvalues. 

Proof. The decreasing convergence case is handled by Theorem [2j Therefore, we only need to construct 
a matrix A with a spectrum A and an initial residual vector ro, ||ro|| = /(0) for which restarted GMRES 
stagnates at cycle s + 1 while ||n|| = /(l) > . . . > |r s || = /(s), s < q. 

By Lemma[TJ given the initial residual vector ro, ||ro|| = /(0), we can construct residual vectors r& with 
the prescribed norms f(k), and orthonormal sets Wm , k = 1, . . . ,s, such that the set 

{r ,w£l 1 ,...,r B - lt w£_ 1 ,r a } (18) 

is linearly independent. In order to enforce stagnation at the (s + l)-st GMRES(m) cycle, we want the next 
orthonormal set Wm to be orthogonal to the residual vector r s . (See Eq. ([6]) or ([7]).) Thus, following the 
pattern in Lcmma[IJ we choose Wm from the orthogonal complement of the set ([18]) . and append W^-i* 
to the set (fT8"|). thus obtaining the linearly independent set 

{r , 15 • ■ • , r._i, W%_ 1} r a , W^}. (19) 

At this point, if we followed the proof of Lcmma[l] we would append the new residual vector r s+ i to the 
set ([T9"|) . Since r s = r s+ i, this would result in the loss of the linear independence of our set. Instead, we 
would like to expand the set (|19[) by some vector that will not spoil the linear independence and will allow 
for a proper definition of the operator A at the second step of the proof. To fulfill this task, we choose this 
vector to be Wm + r s and append it to (fll?)) . We obtain the set 

{r , Wi 1 ^, . . . , r a -i,W%l 1 ,r a ,wi*¥,wk+ 1 '> + r s }, (20) 

which is linearly independent, since the vector + r s has the component Wm +1 ^ from the orthogonal 

complement of (|19[) and hence cannot be represented as a linear combination of vectors in this set. 
Expanding (|20|) with vectors <S = {si, . . . ,s*}, we finally construct the basis of C n : 

S = {ro, W£l u . . . , r a _i, W^i, r„ + r„ Si, . . . , a*}, (21) 

where t = n — m(s + 1) — 1. 

Now, following the previously described pattern, we need to define an operator A with a prescribed 
spectrum A, represented by the matrix A in the canonical basis, such that Eq. © is satisfied for k = 
1, . . . , s + 1. We split A into the disjoint subsets A = {Ax, A2, . . . , A s+ i, A s+ 2}, so that each A^ consists of 
m sequential elements of A, k — 1, . . . , s + 1, while the rest n — m(s + 1) elements are included into A s+ 2. 
Similarly to (fl3|) - (fT4]) , for each k, we introduce the polynomials 

rn — 1 

Pk (x) = Y. n /' * = !,.••,«+ 1; (22) 

t 

p s+2 {x) = x t+1 -^aj s+2 V, f = n-m(s + l)-l; (23) 
3=0 

where the roots of each polynomial are in the respective set Afc, k = 1, . . . , s + 2. 
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Similarly to (fT5|) . we define the operator A as following: 

Ar k -i = w[ k \ 
Aw[ k) = w (k \ 



• /iu 'm-2 — w m-li 
A (fc) O) , (fc) i (fe) (*) , , (fc) (*-') ; 1 



■4? 



,.(«+!) 



(24) 



• Alu; 7n-2 — w m-l ' 

A + 1) («+!)/- I's+l) , \ . (s + 1) (s+1) (s+1) , (s+1) (s+1) 

A(wt +1) +r s ) = s u 
Asi = s 2 , 

As t -i = s t , 

As t = ai s+2 \w^ +1) +r s ) + a[ s+2 % + --- + a{ s+ Vs u 

where ct~ k s are the coefficients of polynomials ([22]) and (|23p . From the definition (|24|) of the operator A, 
one can observe that for each k, k = 1, . . . , s + 1, 

span{^lr fc _i, . . . , A°rk-i] = span j = 1, , . . ,m. 

Thus, given the representation A of the operator A in the canonical basis, we can guarantee that Eq. ([2]) 
holds for each fc, k = 1, . . . , s + 1. 

Similarly to Eq. (fl6|) . the structure of the matrix [A]g of the operator A in the basis iS will be block 
lower triangular with each diagonal block being the companion matrix for the corresponding subset of 
A's eigenvalues, where characteristic polynomials are defined by (j22"]) - (|23)) , and — a^'s being subdiagonal 
elements. The desired matrix A is then obtained by similarity transformation 

A = S[A\ s S-\ 

where the square matrix S is formed by the set of vectors ([2Tj) written as columns. 

□ 



4 Restarted GMRES with variable restart parameter 

The result given by Theorem [3] generalizes to the case when the restart parameter m is not fixed, but varies 
over the successive cycles with a priori prescribed restart parameters m k for the corresponding GMRES (wfc) 
cycles. 

Corollary 1 Given a matrix order n, a sequence {rrik\ q k=1 of restart parameters with 1 < rrik < n — 1, 

9 

< n, and a positive sequence {f{k)}1 =Q , which is either decreasing, or such that /(0) > /(l) > • • • > 

k=l 

f(s) > and f(s) = f(s + 1) = . . . = f{q), where s < q. There exits an n-by-n matrix A and a vector ?*o 
with \\ro\\ = /(0) such that \\rk\\ = f{k), k = 1, . . . , q, where r k is the residual at cycle k of restarted GMRES 
with a variable restart parameter applied to the linear system Ax ~ b, with initial residual rg = b — Axq . 
Moreover, the matrix A can be chosen to have any desired eigenvalues. 
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Proof The proof follows directly from Lemma [TJ Lemma and Theorem [5] Note that the constructed 
operator A will have block lower triangular matrices with block sizes mk (instead of m). 

□ 



5 Generating non- convergent examples 

When constructing a matrix A and an initial residual vector ro which provide the prescribed decreasing 
cycle-convergence generated by GMRES(m), we note that from the last line of the definition (fT"5|) of the 
operator A we obtain 

r q g AKt+i {A,r q ) , 

where A is the representation of the operator A in the canonical basis and t = n — mq — 1 . This equality 
implies that at the end of the (q + l)-st cycle GMRES(m) converges to the exact solution of Eq. JJJ), i.e. 
r q+ i = 0. This fact might seem unnatural and undesirable, e.g., for constructing academic examples. The 
"drawback" , however, can be easily fixed by a slight correction of the basis S - somewhat similarly to how 
we handled the stagnation case in Theorem [3J 

Given residuals r k and orthonormal sets Wm constructed according to Lemma [TJ instead of considering 
the set S, we consider the following basis of C": 

S = {r ,w^\ . . • . . .,r q -i,w\ q) ,. . •,w„_i>'"<2 + jr q - 1 ,s 1 , . . . ,s t }, (25) 

where 7 ^ -1. Here we substituted the basis vector r q in Eq. (|12p by r q + jr q —x. The vector r q + jr q -i 
cannot be represented as a linear combination of other vectors in S, since it contains the component r q , 
which is not represented by these vectors. Hence, S is indeed a basis of C™. Thus we can define the operator 
A by its action on S: 

Ar k ^.i = w[ k >, 
Aw[ k} = wf\ 



(*>) „.(*) 

m — 2 rn — 

( fc ) - „S k K _L™( fc )„ ^^,( fc ),„( fc )_L _L„,( fe ) „„( fc ) 



Ai 



q-l 



.,('/) 



(26) 



Aw[ q} = w (q \ 



^ w m~2 — w m-li 
(<j) 

Awit, = T ^(r g + 7r ! -i)+^ ) r,-i + «W ) + - + ^i> 
1 + 7 

A(r q +7r 9 _i) = si, 
As 1 = s 2 , 



A&t-x = s t , 

As t = ^ +1 \r q+1 r q _ l ) + ^ +1 h l + --- + a\ q+1) s u 

where oq 's are the coefficients of the corresponding characteristic polynomials (fl3|) and (|14p . The fact that 
the operator A produces the correct Krylov residual subspace at the cycle q, i.e., 

span{^r 9 _i, . . . , A m r q -i\ = span W$ , 
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can be observed from the following equalities: 

(?) 



1 


+ 7 




a (9) 




"0 


1 


+ 7 




a {q) 




"0 


1 


+ 7 



m-l w m-l 



Hq r 9-l+ Q l w l + rfl m _lVl 



where, by Eq. (26]), -Aw^Li = ^ m r 9 _i and, by Eq. J7J), (r g - r q - X ) G span{w^ } }. 

The matrix [.A]j of the operator .4, defined by Eq. I|26[). in the basis iS is identical to Eq. (fl"6)) with the 

only change of the subdiagonal element — a^f 1 to -j^-, 7^ -1. Hence, A has the desired spectrum A. 
Thus, finally, according to Eq. (f2"6"| . 

r 9 6 -4JC t+ i (y4, r g ) + /C t+2 (-4, r g _i) , 

providing that r q+ \ is nonzero. 
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